Blow tracking user interface system and method

ABSTRACT

A blow tracking user interface method and apparatus may detect an orientation of blowing of a user&#39;s breath and a magnitude of blowing of the user&#39;s breath. A blow vector may be generated from the orientation and magnitude of the blowing of the user&#39;s breath. The blow vector may be used as a control input in a computer program.

FIELD OF THE INVENTION

Embodiments of this invention are directed to user interfaces forcontrol of computer systems and more specifically to user interfacesthat track the blowing of a user's breath to provide control input to acomputer program.

BACKGROUND OF THE INVENTION

There are a number of different control interfaces that may be used toprovide input to a computer program. Examples of such interfaces includewell-known interfaces such as a computer keyboard, mouse or joystickcontroller. Such interfaces typically have analog or digital switchesthat provide electrical signals that can be mapped to specific commandsor input signals that affect the execution of a computer program.

Recently, interfaces have been developed for use in conjunction withvideo games that rely on other types of input. There are interfacesbased on microphones or microphone arrays, interfaces based on camerasor camera arrays. Microphone-based systems are used for speechrecognition systems that try to supplant keyboard inputs with spokeninputs. Microphone array based systems can track sources of sounds aswell as interpret the sounds. Camera based interfaces attempt to replacejoystick inputs with gestures and movements of a user or an object heldby a user.

Different interfaces have different advantages and drawbacks. Keyboardinterfaces are good for entering text but less useful for enteringdirectional commands. Joysticks and mice are good for enteringdirectional commands and less useful for entering text. Camera-basedinterfaces are good for tracking objects in two-dimensions but generallyrequire some form of augmentation (e.g., use of two cameras or a singlecamera with echo-location) to track objects in three dimensions. Suchaugmentation can increase the cost of a camera-based system.

It would be desirable to provide an interface that is intuitive to useand is also relatively inexpensive to implement.

It is within this context that embodiments of the present inventionarise.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present invention can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings, in which:

FIG. 1 is a flow diagram illustrating a blow tracking interface methodaccording to an embodiment of the present invention.

FIGS. 2A-2B are schematic diagrams illustrating use of a blow trackinginterface according to an embodiment of the present invention.

FIG. 3A is a schematic diagram illustrating use of a blow trackinginterface according to an alternative embodiment of the presentinvention.

FIG. 3B is a schematic diagram illustrating use of a blow trackinginterface according to another alternative embodiment of the presentinvention.

FIG. 3C is a schematic diagram illustrating use of a blow trackinginterface according to another alternative embodiment of the presentinvention.

FIG. 4 is a block diagram depicting an example of a computer implementedapparatus that uses a blow tracking interface in accordance with anembodiment of the present invention.

FIG. 5 is a block diagram illustrating a non-transitory computerreadable medium containing computer-readable instructions forimplementing a blow tracking interface method according to an embodimentof the present invention.

DESCRIPTION OF THE SPECIFIC EMBODIMENTS

Embodiments of the present invention implement a new user interface forcomputer programs based on the detection of blowing of the user's breathto provide a control input.

INTRODUCTION

The blowing of breath from either the mouth or nose is probably one ofthe easiest and most instinctive controls that a person possesses afterbirth. The history and the thermal residue of the breath can also be animportant input for a user interface. One of the easiest and most costeffective ways to estimate information regarding the blowing of a user'sbreath can be achieved is through use of a thermal imaging camera. Inembodiments of the present invention, the direction, timing, strength,and source location of the blowing of breath can be used to control auser interface or to control a graphic display, sound playback, or othermechanical or virtual device. The blowing of user's breath can serve asa kind of laser pointer without requiring the user to hold any externaldevice by hand or attach any device to the user's body. The direction ofthe blowing of breath can be estimated from a thermal trail when oneblows the air toward an arbitrary direction, which need not necessarilybe perpendicular to the user's face. The direction of the blowing ofbreath can also mean whether the user breathes in or breathes out.Breath flow control is very important for learning singing, for mindmeditation, or for expressing emotion. For example, in a video gameimplementation, the direction and intensity of flow of a user's breathcan be used as a trigger for the direction and power of an avatarmonster's breath power. The control of the breath of virtual creaturesin the games with seemly no effort at all on the part of a player willhave a profound impact on the perception and playability of the games.

EMBODIMENTS

Examples of embodiments of the invention may be understood by referringsimultaneously to FIG. 1 and FIGS. 2A-2B. FIG. 1 illustrates an exampleof a blow tracking interface method 100 in which the blowing of a user'sbreath is used to provide a control to a computer program. Specifically,the blowing of a user's breath is detected as indicated at 102. Thereare a number of ways in which the blowing of a user's breath may bedetected. By way of example and not by way of limitation, FIG. 2Aillustrates a possible system 200 that may be used to implement a breathtracking interface. The system generally includes a thermographic camera202 coupled to a computer processing system 204. The thermographiccamera may be positioned proximate a video display 214 that is coupledto the processing system 204 such that the user faces the camera whenfacing the display.

Use of a thermographic camera to track the blowing of a user's breath isparticularly advantageous because it avoids the need to put sensors onbody, e.g., to track the orientation of the user's head. This isparticularly true since the orientation of the nose with respect to theface is substantially fixed. Although the shape of the mouth can bechanged voluntarily in a relatively short time it is much harder tochange the shape of the nose or the orientation of the nose relative tothe face. Consequently the nose breath vector 212 tends to blow in afixed direction with respect to the face. Consequently, the nose breathvector can be used to determine the orientation of the user's face.

Blow detection using a thermographic camera or similar sensor is alsoadvantageous in that it avoids the need for the user to hold acontroller device by hand. This frees the user's hands for other tasks.

As used herein, a thermographic camera refers to a type of camerasometimes called a Forward Looking Infrared (FLIR) or infrared camerathat forms an image using infrared radiation. The thermographic camera202 forms an image using one or more lenses and a sensor in a fashionsimilar to a camera that forms an image using visible light. Instead ofthe 450-750 nanometer range of the visible light camera, an infraredcamera can operate in wavelengths as long as 14,000 nm (14 μm).

The thermographic camera 202 can take advantage of the fact that allobjects emit a certain amount of black body radiation as a function oftheir temperatures. Generally speaking, the higher an object'stemperature, the more infrared radiation the body emits as black-bodyradiation. An infrared camera can detect this radiation in a way similarto an ordinary camera does visible light. However, since bodies emitinfrared radiation even in total darkness, the ambient light level doesnot matter.

Embodiments of the present invention may use more than one thermographiccamera. For example, two thermographic cameras may be used in aside-by-side configuration to provide stereoscopic (3D) images that canprovide three-dimensional information. An equivalent thermographic“stereo” camera may be implemented in a single device havingside-by-side lenses that image different views of the some scene onto asensor or array of sensors. Multiple cameras might be used to create athree-dimensional representation of the breath volume and rate of flow(i.e., volume of breath flowing per unit time).

The thermographic camera 202 may use a cooled thermal sensor or anuncooled thermal sensor operating at ambient temperature, or a sensorstabilized at a temperature close to ambient using small temperaturecontrol elements. Modern uncooled detectors use sensors that detectchanges of resistance, voltage or current when heated by infraredradiation. These changes can then be measured and compared to values atthe operating temperature of the sensor. Uncooled infrared sensors canbe stabilized to an operating temperature to reduce image noise, butthey are not cooled to low temperatures and do not require bulky,expensive cryogenic coolers. This makes infrared cameras smaller andless costly.

Uncooled detectors are mostly based on pyroelectric and ferroelectricmaterials or microbolometer technology. These materials are used to forma detector having an array of pixels with highly temperature-dependentproperties. These pixels can be thermally insulated from the surroundingenvironment and read electronically.

Ferroelectric detectors operate close to a phase transition temperatureof the sensor material. The pixel temperature of the detector iscalibrated to a highly temperature-dependent polarization charge on thedetector material. The achieved noise equivalent temperature difference(NETD) of ferroelectric detectors with f/1 optics and 320×240 sensorscan be 70-80 mK. An example of a possible ferroelectric sensor assemblyconsists of barium strontium titanate bump-bonded by polyimide thermallyinsulated connection. Other possible phase-change materials that can beused in infrared detectors include lanthanum barium manganite (LBMO), ametal insulator phase change material.

There is another possible detector that can detect small changes in theelectrical resistance of a sensor material. Such a device is sometimescalled a microbolometer. Microbolometers can reach NETD down to 20 mK. Atypical microbolometer includes a thin film vanadium (V) oxide sensingelement suspended on silicon nitride bridge above the silicon-basedscanning electronics. The electric resistance of the sensing element canbe measured once per frame.

It is noted that the location of the thermographic camera 202 issomewhat arbitrary in the example shown in the FIG. 2A. Thethermographic camera 202 is placed on top of a video display facing auser. However, embodiments of the present invention are not limited tosuch a configuration.

The values of temperature (or temperature difference) at each pixel ofthe image sensor in the thermographic camera can be stored in a memoryof the computer system 204 as an array. A program executed on thecomputer system 204 can analyze the temperature patterns in the array toidentify a user's face and to identify thermal patterns that correspondto the blowing of a user's breath, e.g., by analyzing the evolution ofthermal contours over time at locations corresponding to a user's mouthM and nose N. FIG. 2B illustrates an example of a thermal image showingthermal contours 206, 208 that correspond to the blowing of breath fromthe user's mouth and nose, respectively. In some embodiments, multiplesynchronized thermal images from multiple cameras can be used to improvethe resolution and location of the breath. The computer system 204 mayanalyze the evolution of the thermal contours 206, 208 over time todetermine whether they exhibit patterns characteristic of a user'sbreath. Such characteristics may include the shape and size of thecontours, the timing of changes in the contours, the gradient (orsteepness) of the contours, and the location of the contours relative tofeatures on the user's face, such as the mouth M and the nose N.

Such analysis can determine a timing of the blowing of breath, asindicated at 104. The timing of the blowing may include determining whena breath starts or stops, the duration of a breath, or the duration ofnot breathing, e.g., an interval between breaths. In some embodiments,the apparatus 200 may include an optional auxiliary sensor, such asmicrophone or microphone array 205 to provide sound signals that thecomputer system 204 can correlate to the blowing of the user's breath.In some embodiments, the microphone/microphone array 205 can be worn bythe user. For example, the user may wear a headset having a boom towhich the microphone 205 is attached. The boom can be configured so thatthe microphone 205 hovers right in front of the user's mouth and nose.

In some embodiments a second thermographic camera or conventional (i.e.,visible light) camera, may be used as the auxiliary sensor to provide anadditional source of information, e.g., thermographic, acoustic, orvideo image information, regarding the blowing of breath by the user. Byway of example, and not by way of limitation, a separate one-dimensionalor two-dimensional thermographic camera 202A may be mounted to a pair ofglasses 203 worn by the user (e.g., a pair of 3D shutter glasses). Theseparate (additional) thermographic camera can obtain additionalthermographic information related to the blowing of breath from theuser's point of view. This additional thermographic information can becorrelated to information from the main thermographic camera (or viceversa) to refine calculations regarding the timing, magnitude ororientation of blowing of breath by the user.

The thermal contours 206, 208 can be used to determine other informationabout the user's breath that can be used to control a program running onthe computer system 204. By way of example, and not by way oflimitation, an orientation of the blowing of breath may be detected, asindicated at 106 and a magnitude of the blowing of breath may bedetected as indicated at 108. The orientation and magnitude can be usedto generate a blow vector, as indicated at 110. The blow vector can thenbe used as a control input to a computer program running on the computersystem 204 as indicated at 112.

By way of example, and not by way of limitation, detecting theorientation of breath at 106 may include determining a direction ofbreath, e.g., whether the breath is directed inward or outward (i.e.,whether the breath is an inhale or exhale), a source of the breath,(e.g., whether the breath is emitted through the user's nose, mouth, oranother part of the body or through an object such as a straw held tothe user's lips. It is noted that the vector 110 is not limited to a3-dimensional geometric vector. The vector 110 may include additionalinformation regarding the blowing of breath. Such information mayinclude information about the source of the breath, e.g., the nose N orthe mouth M. For example, the vector 110 may include a component thatcharacterizes the shape of the breath from an analysis of the contours206, 208. A thin shape of air flow might indicate a whistle. There are anumber of different ways to determine the magnitude of the blowing ofbreath. For example, the time evolution of the thermal contours may beanalyzed to estimate the volume of air inhaled or exhaled in one breath,the speed of flow of one air during exhale or inhale, or by the amountof heat associated with a breath. Furthermore, some combination of airvolume, air speed, or heat associated with breath may be used todetermine the magnitude of the blowing vector 110.

In some embodiments the characteristics of the users mouth M can be usedto indicate an orientation (e.g., direction) of the blowing vector 110.For example, a wide open mouth as opposed to pursed lips might indicatedifferent blowing orientations. Information regarding the configurationof the user's mouth may be determined from analysis of the timeevolution of thermal contours in thermographic images. The analysis maycompare information derived from the contours (e.g., dimensions ofcontours, etc.) to reference information derived from thermographicimages in which users' mouths are in known orientations.

In some embodiments the computer system 204 can be programmed todetermine an intended “temperature” of the breath as possibleorientation or magnitude of the breath. For example, when trying to warmthe hands by blowing on them a person uses a heavy throaty breath fromdeep inside with a relatively open mouth. If the person is trying tocool soup by blowing on it, the person might blow gently through pursedlips. These different blowing configurations can be distinguished fromanalysis of the thermographic images to determine mouth orientationand/or breath flow rate. Images from a normal (i.e., visible light)camera can be used to help decide the intended “temperature” of thebreath by providing some additional cues of the shape of the lip whenthe blowing happens. Furthermore, since these different types of blowingtend to have different characteristic sounds, it is also possible todetermine these different intended breath “temperatures” from analysisof acoustic signals obtained with the microphone 205.

Another possibility is to measure the length of a trail of the blowingof breath from the contours 206, 208 and use that as an input forestimating the magnitude of the vector 110. Furthermore, the soundamplitude associated with the blowing of breath can also provide aninput for estimating the magnitude of the vector 110. In certainembodiments, the contours 206, 208 may be analyzed to determinelocalized gradients at different places on the contours. In addition,the steepness of the thermal contours 206, 208 may be analyzed atdifferent locations to determine not just the direction but also themagnitude of the localized gradients. Localized gradient vectors 210 canbe generated from the directions and magnitudes of the localizedgradients. The localized gradient vectors 210 may be averaged orotherwise combined to determine a general breath vector 212 indicatingthe magnitude and direction of the user's breath relative to the user'sface. It is noted that separate general breath vectors 212 couldgenerated for breath at the mouth and nose. These separate breathvectors may be used as separate control inputs for different purposes.In some embodiments, long term history information regarding thecontours 206, 208, localized gradients, localized vectors 210, orgeneral vector 212 may b e used to improve the calculation of themagnitude, location, or orientation of the blowing of breath. Such longterm history may be captured over some finite number of previous breathsor some finite period of time, e.g. 1-30 seconds.

There are a number of different ways in which a breath control inputsuch as a breath vector 212 may be used in a computer program. Ingeneral, the program may change its state of execution in response to adetermined value of the orientation, magnitude or timing of the blowvector. By way of example, and not by way of limitation, the start ofexecution of a program or a subroutine within a program may be triggeredwhen the orientation, magnitude or timing above or below a thresholdvalue. Alternatively, the magnitude, orientation, or timing of the blowvector may determine the value of a variable that is used by theprogram.

In a more general sense, the orientation, magnitude or timing of theblow vector may serve as inputs that control actions resulting fromexecution of the program. For example, a breath control vector can beused in a video game to direct a simulated projectile, e.g., as in afirst person shooter game. Alternatively, the direction and magnitude ofthe breath can be used in a highly intuitive game situation, such asblowing out a candle, as seen in FIG. 3A. In this example, theorientation of the user's breath is determined relative to a videoscreen 214. It is also possible to use the relative change in directionof the blowing of breath from one breath to provide a directional inputwithout reference to the video screen 214. This might be useful, e.g.,in the case of a thermographic camera mounted to a pair of glasses wornby the user. The orientation of the user's breath relative to the videoscreen can be determined if the position and orientation of the camera202 are known in relation to the position and orientation of the videoscreen 214. This can be accomplished, e.g., using a position sensorattached to the camera 202 that senses the position and orientation ofthe camera relative to the video screen. Given the direction of thebreath and location of the user's mouth and/or nose relative to thevideo screen 214, it is easy to implement game play involving gunshooting or candle blowing using breath control. Furthermore, becauseone can blow to multiple directions, the functions of a conventionalgame controller's buttons and/or joystick(s) can be easily implementedwhile leaving the user's hands free.

Breath control can also be used to control the animation of avatars. Acomputer game program may map the timing and intensity of breathing tobreath to an avatar controlled by a user so that avatar breathes insynchronization with user. Such an effect can enhance the realism of thegame experience for the user.

Breath-based control of avatars may also be used in more fanciful ways.For example, as depicted in FIG. 3B, a user's breath can be used in avideo game in which the user controls a dragon avatar. The user'sbreathing can be used to synchronize certain actions of the dragon. Forexample, when the user breathes out with sufficient strength through themouth and/or nose, the game program can trigger dragon's nose to emitout flames or smoke or some other special effect. In some embodiments, auser can customize the breath display pattern for an avatar topersonalize the avatar in a game that is played over a network.

Another possible game implementation is a swimming game in which a userattempts to hold his breath while the user's avatar is underwater.

Embodiments of the invention may use a conventional keyboard or gamecontroller in conjunction with a blow detector that detects the blowingof breath from a user to control multiple functionalities in aneffortless fashion. In some embodiments a blow detector may be combinedwith a conventional input device. For example, a thermographic cameracan be mounted to a keyboard or hand held game pad to provide breathinformation.

Embodiments of the present invention provide tremendously intuitive andversatile control for interactive computer applications. Control overthe timing, strength, and direction of breath is highly intuitive.Furthermore, because one can “point” one's breath in directions otherthan perpendicular to one's face, breath-based controller's can behighly versatile. For example, one can stick out the lower lip or skewthe low lip in order to blow the air upwards on one side of the face. Inthe example shown in FIG. 3C, this can be detected with a thermalimaging camera and mapped to an avatar. The direction and intensity ofthe breath determined from the thermal imaging can control a simulationin which the hair of avatar can be blown upwards or a fly in front ofthe avatar's forehead can be blown away.

Breath detection can also be used to provide feedback in exercise ormeditation related applications. For example, control of breathing is animportant feature of many forms of exercise and mediation, such asChinese qi gong, or vocal breathing exercise for singing. A computerapplication that teaches these techniques may display an avatar to theuser while detecting the timing, orientation, and magnitude of theuser's breath. The avatar's chest may rise and fall in synchronizationwith the user's breathing providing important visual feedback to theuser.

Breath tracking during sleep can be used to control playing of music toenhance sleeping and to keep monitor user's health. For example, fastermusic can be played to encourage heavier breathing. The blow vector 110can be used in conjunction with computer programs that monitor andcontrol snoring. The detection of the blowing of breath of a sleepinguser can augment the detection of sounds captured with a microphone tofilter out sounds that resemble snoring but are not related to theblowing of breath. Analysis of the information used to generate theblowing vector 110 can also provide insights into the severity andnature of the user's snoring. The richness of information used togenerate the breath vector can also be used to help a doctor to diagnosea patient over a network, e.g., with graphics only.

It is noted that embodiments of the invention can be used for thediagnosis and treatment of sleeping disorders such as sleep apnea thatare related to irregular breathing during sleep. Typically, thediagnosis of sleep apnea requires a patient to sleep at a sleeprehabilitation facility with many sensors attached to the patient'sbody. Use of a non-invasive means of breathing detection, such as theblow detection system described herein can allow the patient to undergodiagnosis at home.

FIG. 4 illustrates a block diagram of a computer apparatus 400 that maybe used to implement picture decoding as described above. The apparatus400 generally includes may include a processor module 401 and a memory402. The processor module 401 may include one or more processor cores.As an example of a processing system that uses multiple processormodules, is a Cell processor, examples of which are described in detail,e.g., in Cell Broadband Engine Architecture, which is available onlineathttp://www-306.ibm.com/chips/techlib/techlib.nsf/techdocs/1AEEE1270EA2776387257060006E61BA/$file/CBEA_(—)01_pub.pdf, which is incorporated hereinby reference.

The memory 402 may be in the form of an integrated circuit, e.g., RAM,DRAM, ROM, and the like). The memory may also be a main memory that isaccessible by all of the processor modules 401. In some embodiments, theprocessor module 401 may local memories associated with each core. Aprogram 403 may be stored in the main memory 402 in the form ofprocessor readable instructions that can be executed on the processormodules 401. The program 403 may include instructions configured toimplement a breath tracking interface, e.g., as described above withrespect to FIG. 1. In particular the program instructions may beconfigured to detect orientation and magnitude of the blowing of abreath, generate a blowing vector from the orientation and magnitude,and using the vector as a control input in the computer program 403 orsome other program executed by one or more of the processor modules 401.The coder program 403 may be written in any suitable processor readablelanguage, e.g., e.g., C, C++, JAVA, Assembly, MATLAB, FORTRAN and anumber of other languages. During execution of the coder program 403,portions of program code and/or data 407 may be loaded into the memory402 or the local stores of processor cores for parallel processing bymultiple processor cores.

Input data 407 may be stored in the memory 402. By way of example, andnot by way of limitation, the input data 407 may include datarepresenting signals from a blow detector 418.

The apparatus 400 may also include well-known support functions 410,such as input/output (I/O) elements 411, power supplies (P/S) 412, aclock (CLK) 413 and cache 414. The apparatus 400 may optionally includea mass storage device 415 such as a disk drive, CD-ROM drive, tapedrive, or the like to store programs and/or data. The device 400 mayalso optionally include a display unit 416 and the blow detector unit418 and user interface 419 to facilitate interaction between theapparatus 400 and a user. The display unit 416 may be in the form of acathode ray tube (CRT) or flat panel screen that displays text,numerals, graphical symbols or images.

The blow detector unit 418 includes a device that is sensitive to theorientation, magnitude, and (optionally) timing of a user's breath. In apreferred embodiment, the breath detector unit may include athermographic camera, which may operate as described above with respectto FIG. 1 and FIGS. 2A-2B. The values of temperature (or temperaturedifference) at each pixel of an image sensor in the thermographic cameracan be stored in the memory 402 as an array. The program 403 can analyzethe temperature patterns in the array to identify a user's face and toidentify thermal patterns that correspond to a user's breath, e.g., bydetermining thermal gradients. Thermal gradients can be used todetermine a breath vector, which may be used as a control input.

The system 400 may optionally include one or more audio speakers 409that are coupled to the processor 401 via the I/O elements 411. Thespeakers can play sounds generated in response to signals generated byexecution of the program 403. The audio speakers 409 can be used, e.g.,when the breath vector triggers the play back of sounds. In someembodiments, the system 400 may include an optional microphone 417,which may be a single microphone or a microphone array. The microphone417 can be coupled to the processor 401 via the I/O elements 411. Theprogram 403 can use sounds of a user's breath to augment the informationfrom the blow detector 418 when determining the magnitude, direction, ortiming of the blowing vector.

The user interface 419 may optionally include a keyboard, mouse,joystick, light pen, microphone, conventional digital camera,accelerometer, gyroscope, or other device that may be used inconjunction with the breath detector unit 418. The apparatus 400 mayalso include a network interface 422 to enable the device to communicatewith other devices over a network, such as the internet. Thesecomponents may be implemented in hardware, software or firmware or somecombination of two or more of these. The breath vector information canbe saved to memory and sent over a network to a doctor for diagnosis viathe network interface 422.

According to another embodiment, instructions for carrying out a blowtracking interface method may be stored in a computer readable storagemedium. By way of example, and not by way of limitation, FIG. 5illustrates an example of a non-transitory computer-readable storagemedium 500 in accordance with an embodiment of the present invention.The storage medium 500 contains computer-readable instructions stored ina format that can be retrieved and interpreted by a computer processingdevice. By way of example and not by way of limitation, thecomputer-readable storage medium 500 may be a computer-readable memory,such as random access memory (RAM) or read only memory (ROM), a computerreadable storage disk for a fixed disk drive (e.g., a hard disk drive),or a removable disk drive. In addition, the computer-readable storagemedium 500 may be a flash memory device, a computer-readable tape, aCD-ROM, a DVD-ROM, a Blu-Ray, HD-DVD, UMD, or other optical storagemedium.

The storage medium 500 contains blow tracking instructions 501configured to implement a blow tracking interface. The breath trackinginstructions 501 may be configured to implement breath tracking inaccordance with the methods as described above with respect to FIG. 2Athrough FIG. 3C. Specifically, the breath tracking instructions 501 mayinclude breath detection instructions 503 that detect blowing of abreath from a detector signal when executed on a processing device. Thebreath tracking instructions 501 may include breath timing determinationinstructions 505, breath orientation determination instructions 507, andbreath magnitude determination instructions 509. When executed, theseinstructions respectively cause the processor to analyze the signal fromthe input device to determine the timing, orientation and magnitude fromthe detector signal.

The blow tracking instructions 501 may further include breath vectorgeneration instructions 511 that generate a breath vector from thetiming, orientation, and magnitude of the breath as determined by thebreath timing determination instructions 505, breath orientationdetermination instructions 507, and breath magnitude determinationinstructions 509. The breath tracking instructions 501 may optionallyinclude control instructions 513 that provide a control input to acomputer program based on the breath vector.

There are many different possible ways of implementing and utilizing ablow tracking interface in addition to the ones described above. Forexample, according to certain alternative embodiments of the presentinvention may be applied to games that involve singing. Certain gamesexist in which a user sings along with a song that is played by acomputer system. A microphone captures the sound of the user's voice. Acomputer program analyzes the sounds of the user's voice for singing ofcertain notes in the song by comparing the pitch of the voice to areference pitch for the relevant note. However, matching pitch is notthe only way, or even the best way to judge singing. Embodiments of thepresent invention can augment or even supplant judgment of singing basedon pitch by following the timing of the user's breathing during singing.The computer program can analyze a signal from a microphone or IR camerato determine the timing and depth of the user's breathing. The timing ofthe user's breathing can be detected for both the inhale and the exhaleduring singing. The program can then compare the timing of the user'sbreathing against a reference breathing pattern for a given song toprovide better evaluation of the singing scores. The program can alsoinstruct a user when to exhale and when to inhale during singing.

Although the above-described embodiments utilize a thermal imagingcamera to detect breath, embodiments of the invention encompass otherways to detect breath, such as with a microphone or by observing thedeflection of sensor, such as a feather or piece of tissue paper with avideo camera. Consequently, embodiments of the present invention are notlimited to the use of a thermal imaging camera to detect breath.Alternatively, embodiments of the present invention may use otherapparatus, such as one or more microphones to detect breath, e.g., as ajoystick. For example, a user might wear a headset having a microphoneto detect the timing and amplitude of the user's breath the headsetcould also include an inertial sensor, such as an accelerometer, todetect the orientation of the user's head. The orientation of the user'shead can be used to determine the direction of the breath vector andsignals from the microphone can be filtered to isolate breath sounds.The timing and magnitude of the breath vector can be determined fromanalysis of the breath sounds.

In some embodiments, breath detection may be used for computer programsecurity. Breath detection can confirm that a verbal passcode is beingspoken by person speaking it. This would make it more difficult todefeat a verbal passcode system, e.g., through playback of a recordingof a user speaking the passcode.

Embodiments of the present invention can also use a thermal imagingcamera or other breath detector to augment speech detection. A change ina user's breathing pattern can be used to determine when the user isspeaking. It is also to use breath detection in a crowd of users (e.g.,by using a thermal imaging camera) to detect which user in the crowd isspeaking.

In other embodiments, one can use a thermal imaging camera or othermeans to detect a user's breath and to subtract the sounds of the breathfrom a microphone signal to reduce speech noise.

In further embodiments it is possible to use a thermal imaging camera todetect vibration patterns in the air around a user's mouth thatcorrespond to the sounds of the user's voice during speech. Thevibration patterns can be analyzed used to augment or supplant speechrecognition based on speech input signals from a microphone.

Although the present invention has been described in considerable detailwith reference to certain preferred versions thereof, other versions arepossible. For example, although certain embodiments are described inwhich the breath tracking interface generates a breath vector from amagnitude and direction of a breath, embodiments of the invention mayinclude implementations in which the breath tracking is implementedaccording to some other method, e.g., one not involving magnitude of thebreath or not involving orientation of the breath. In addition, certainembodiments may rely upon timing of the breath independent of directionor magnitude to provide a control input. Therefore, the spirit and scopeof the appended claims should not be limited to the description of thepreferred versions contained herein. Instead, the scope of the inventionshould be determined with reference to the appended claims, along withtheir full scope of equivalents.

All the features disclosed in this specification (including anyaccompanying claims, abstract and drawings) may be replaced byalternative features serving the same, equivalent or similar purpose,unless expressly stated otherwise. Thus, unless expressly statedotherwise, each feature disclosed is one example only of a genericseries of equivalent or similar features. Any feature, whether preferredor not, may be combined with any other feature, whether preferred ornot. In the claims that follow, the indefinite article “A”, or “An”refers to a quantity of one or more of the item following the article,except where expressly stated otherwise. Any element in a claim thatdoes not explicitly state “means for” performing a specified function,is not to be interpreted as a “means” or “step” clause as specified in35 USC §112, ¶6. In particular, the use of “step of” in the claimsherein is not intended to invoke the provisions of 35 USC §112, ¶6.

The reader's attention is directed to all papers and documents which arefiled concurrently with this specification and which are open to publicinspection with this specification, and the contents of any papers anddocuments incorporated herein by reference.

1. A blow tracking interface method for control of a computer program, comprising: detecting orientation of blowing of a breath; detecting a magnitude of blowing of the breath; generating a blow vector from the orientation and magnitude; and using the blow vector as a control input in a computer program.
 2. The method of claim 1, further comprising determining a timing of the blowing of the breath, wherein the timing of the blowing of the breath determines timing of an action in the computer program.
 3. The method of claim 2, further comprising synchronizing timing of actions in the computer program to the timing of the blowing of breath.
 4. The method of claim 1, wherein detecting the orientation and/or magnitude of the blowing of the breath includes obtaining one or more thermal infrared images of a user with one or more thermographic cameras, analyzing the one or more infrared images, and identifying one or more patterns in the one or more images corresponding to the blowing of breath from the one or more thermal infrared image.
 5. The method of claim 4 wherein detecting the orientation and/or magnitude of the breath includes analyzing thermal contours in the one or more thermal infrared images, identifying one or more patterns characteristic of the blowing of the breath from the thermal infrared image, and determining the orientation from the one or more patterns.
 6. The method of claim 4, wherein the thermographic camera is mounted to a pair of glasses worn by the user.
 7. The method of claim 4, wherein generating the blow vector includes augmenting information obtained from the one or more thermal infrared images with information relating to the breath from one or more auxiliary sensors.
 8. The method of claim 6, wherein the one or more auxiliary sensors includes a microphone or camera.
 9. The method of claim 6, wherein the auxiliary detector is an additional thermographic camera mounted to a pair of glasses worn by the user.
 10. The method of claim 1 wherein generating a blow vector includes generating a blow vector for breath through the user's nose and determining an orientation of the user's face from an orientation of the blow vector for breath through the user's nose.
 11. The method of claim 1, wherein detecting the orientation and/or magnitude of the blowing of the breath includes detecting sounds of the blowing of the breath with a microphone.
 12. The method of claim 1 wherein detecting the orientation of the breath includes determining a source of the blowing of the breath.
 13. The method of claim 6 wherein determining the source of the breath includes determining which user from among a plurality of users is the source of the blowing of the breath.
 14. The method of claim 6 wherein determining the source of the breath includes determining whether the source of the blowing of the breath is a user's mouth or nose.
 15. The method of claim 1, further comprising mapping the timing and intensity of a user's breathing to simulated breathing of an avatar controlled by a user so that the avatar appears to breathe in synchronization with user.
 16. The method of claim 1 wherein detecting orientation of a breath includes determining whether a user is blowing breath through the user's mouth or nose.
 17. The method of claim 1 wherein detecting orientation of a breath includes determining whether a user is breathing in or out.
 18. The method of claim 1 wherein detecting orientation or magnitude of a breath includes determining an intended temperature of the breath.
 19. The method of claim 1 wherein using the blow vector as a control input includes using the orientation of the breath to determine a direction of an action taken by the computer program.
 20. The method of claim 1 wherein using the blow vector as a control input includes using the magnitude of the breath to determine the magnitude of an action taken by the computer program.
 21. The method of claim 1 wherein the orientation of the breath is determined relative to a video display.
 22. The method of claim 1 wherein generating the blow vector includes using a past breath history to improve an estimation of the blow vector.
 23. A blow tracking interface apparatus for control of a computer program, comprising: a blow detector configured to detect one or more signals corresponding to blowing of a breath; and a processor coupled to the blow detector, wherein the processor is configured to determine an orientation and magnitude of the blowing of the breath from the one or more signals, and generate a blow vector from the orientation and magnitude, wherein the processor is configured to use the blow vector as a control input in a computer program.
 24. The apparatus of claim 23 wherein the breath detector includes one or more thermographic cameras.
 25. The apparatus of claim 24, wherein the processor is configured to detect the orientation and/or magnitude of the breath by analyzing thermal contours in the image to identify one or more patterns characteristic the breath from the thermal infrared image.
 26. The apparatus of claim 24 wherein the one or more thermographic cameras include a thermographic camera mounted to a video display that is coupled to the processor.
 27. The apparatus of claim 24 wherein the one or more thermographic cameras include a thermographic camera mounted to a pair of glasses.
 28. The apparatus of claim 23 wherein the blow detector includes one or more microphones.
 29. A non-transitory computer readable medium having computer executable instructions embodied therein, wherein the instructions are configured to implement a blow tracking interface method for control of a computer program when executed by a computer processor, wherein the method implemented by the execution of the instructions comprises: detecting orientation of blowing of a breath; detecting a magnitude of blowing of the breath; generating a blow vector from the orientation and magnitude; and using the blow vector as a control input in a computer program. 